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1.  INTRODUCTION  AND  OVERVIEW 

Recent  studies  (ref  [3],  [A],  [5],  [7])  by  the  Inventory  Research  Office  (IRO) 
have  indicated  certain  anomalies  between  methods  of  evaluating  demand  forecast 
algorithms  for  use  in  an  inventory  management  system.  These  studies  were  based 
on  empirical  evidence  gleaned  from  a  data  base  of  quarterly  demands  for  over 
20,000  aviation  repair  parts  spanning  a  history  of  eleven  years.  In  each 
study  both  statistical  and  simulated  performance  measures  were  computed.  (The 
statistics  were  collected  by  comparing  the  forecasts  with  the  actual  values  of 
the  series  whereas  the  simulation  results  revealed  how  well  the  forecasts  per¬ 
formed  in  the  simulated  supply  system.  Both  methods  utilized  actual  demand 
data. ) 

This  note  emphasizes  the  problem  of  inconsistency  by  showing  specific 
examples  where  alternative  measures  of  performance  yielded  different  results. 

In  each  case  a  "best"  forecast  algorithm  was  sought  but  discrepancies  between 
the  error  measures  confounded  the  analysis.  Comparative  ranks  were  made 
using  each  measure  in  an  attempt  to  determine  the  best  algorithm(s) .  It  is 
the  rank  variability  between  the  evaluation  methods  that  is  the  concern  of 
this  report. 

It  was  hoped  that  statistical  error  measures  could  be  found  that  would 
be  consistent  with  the  supply  performance  measures.  By  so  doing,  both  theore¬ 
tical  and  empirical  guidance  would  be  given  for  the  development  of  optimal 
forecasts.  The  examples  cited  herein  trace  the  developmental  stages  of  the 
measures.  The  first  considered  were  simple  statistical  expressions  such  as 
mean  squared  error  (MSE) ,  mean  absolute  error  (MAD)  and  average  error  (BIAS) . 

Based  on  experiences  with  these  basic  measures,  percent  and  relative  measures 
were  developed  along  with  changes  in  processing  such  as  forecast  horizon 
and  forecast  timing.  Next,  inventory  measures  were  developed  where  both  cost 
and  performance  were  measured.  The  IRO  simulator  was  used  to  correlate  the 
above  measures  with  inventory  performance.  Finally  a  surrogate  simulation 
method  was  derived  to  best  measure  inventory  performance. 
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2.  ORGANIZATION 


The  first  seven  sections  of  this  report  give  the  basic  background  details 
needed  to  understand  the  comparative  analysis  presented  in  Section  8.  The 
data  base  common  to  all  the  examples  is  described  in  Section  3  along  with 
the  breakout  of  the  various  subsets  or  stratification  classes  which  were  used. 

In  Section  4  the  general  forms  of  the  forecast  algorithms  are  listed.  Sections 
5,  6,  and  7  describe  the  three  general  evaluative  methods  (statistical  error 
measures,  inventory  error  measures,  simulation  analysis)  and  the  computational 
detail  of  each  measure  used.  The  comparative  analysis  is  presented  in  Section  8. 
Based  on  insights  gained  from  Section  8,  a  surrogate  simulation  method  is 
developed  and  tested  in  Section  9.  In  Section  10  specific  problems  in  evaluating 
series  with  many  zeros  are  mentioned.  The  conclusions  based  on  the  findings  are 
stated  in  Section  11. 


Acre: 

:■  Jon  For 

'  NTT  F 

•  r  W  “ 

T>t: 

r 

□ 

V  ;>. 
Jl:  ..  . 

n 

— 

Dir.tr 

i but:  <•;:)/ 

Availability  Codes  j 

Avail  and/or 

Dist 

flj 

Special 

3 


3 .  DATA  STRUCTURE 


The  data  common  to  all  the  examples  in  this  report  is  from  the  IRO  demand 
history  file  which  includes  11  years  of  requisitions  and  demand  by  quarter 
for  aviation  repair  parts.  The  file  contains  a  sample  of  20,865  items  from 
all  those  in  the  system  in  1966  and  subsequently  entered.  A  more  detailed 
description  as  to  the  type  of  item  is  found  in  [2]  . 

For  the  initial  analysis  each  item  was  classified  as  low  dollar  value  (LDV) 
or  high  dollar  value  (HDV)  according  to  whether  the  demand  rate  averaged  over 
the  11  years  was  less  than  or  greater  than  or  equal  to  $50,000;  and  the 
requisition  rate  was  less  than  or  greater  than  or  equal  to  100  per  year. 

Items  with  over  a  million  dollars  of  demand  per  year  were  dropped. 

The  items  were  further  divided  into  dynamic  (DYN)  and  non-dynamic  (NON) 
based  on  the  Federal  Stock  Class  (FSC) .  The  dynamic  components  were  considered 
to  be  those  that  experience  high  rotation  rates;  i.e.  rotor  blades,  trans¬ 
missions,  and  turbine  engines.  For  more  detail  see  Cohen  [2]. 

The  data  breaks  out  into  the  following  four  groups: 


HDVDYN 

86 

HDVNON 

262 

LDVDYN 

1169 

LDVNON 

19348 

20865 

Later  on,  samples  were  taken  from  the  data  and  stratified  in  classification 
cells  according  to  the  item's  requisition  frequency  and  dollar  demand.  The 
"strat"  class  were  identified  as  follows: 


Yearly  Dollar  Demand 


4 .  FORECAST  METHODS 


The  five  basic  forecast  algorithms  tested  were  of  the  form  of  a  weighted 
moving  average  (1794),  moving  median  (MED),  Kalman  filter  (KAL),  a  variable 
base  moving  average  utilizing  a  tracking  signal  (TRIG)  and  a  Bimple  moving  average 
on  demand  (MOVD) .  The  underlying  structure  of  the  algorithms  are  as  follows: 
(1794) : 

This  is  the  current  Army  forecast  procedure  which  is  a  weighted 
eight  quarter  moving  average  with  weights  proportional  to  quarterly 

flying  hours. 

(MEDIAN) 

This  is  the  median  of  the  last  four  quarters. 

(Kalman  Filter) 

For  a  general  class  of  statistical  processes,  this  is  a  minimum 
mean  squared  error  algorithm  which  is  structurally  similar  to  exponential 
smoothing  where  the  smoothing  weights  are  variable  and  updated. 

(TRIG  Tracking  Signal) 

This  is  a  version  of  the  TRIG  methodology  which  switches  between  a 
weighted  four  and  eight  quarter  moving  average  where  the  weights  are  parameters 
determining  the  amount  of  additional  weight  to  give  the  current  observation. 

(MOVD) 

This  is  an  eight  quarter  moving  average  on  demand. 

See  reference  [4]  for  additional  details. 
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5.  STATISTICAL  ERROR  MEASURES 


In  an  inventory  management  Betting,  an  optimal  forecast  is  one  that  per¬ 
forms  well  over  many  items  each  of  which  may  have  different  forecast  horizons 
(procurement  lead  time).  Each  forecast  error  impacts  the  overall  performance 
of  the  system  by  a  different  magnitude  depending  on  the  demand  activity  of  the 
series.  Various  methods  of  aggregating  these  errors  are  presented  in  this  section. 

For  the  experiment,  one  period  (qtr),  four  period  and  variable  horizons 
were  used  for  comparative  purposes,  but  for  simplicity  the  notation  assumes  a 
one  period  forecast  horizon.  There  is  little  in  the  literature  describing 
empirical  work  comparing  errors  between  various  forecast  horizons.  Peter 
Bloomfield  of  Princeton  is  doing  some  theoretical  work  in  this  area. 

Notation 

Let  x  be  the  demand  in  the  jth  quarter  (for  the  1th  item) 
and  the  forecast 

F  -  index  set  of  forecasts  for  1th  item 

1  i.e.  the  set  of  j  values  for  which  a  forecast  is  made 

•  cardinal  size  of  F^  (number  of  kimci  a  forecast  was  made) 

* 

EtJ  -  error  (x^  -  x^),  Jfc  PJ 

AVG^  -  average  demand  for  item  1 

(Double  12  month  moving  average  starting  after  the  first 
non-zero  demand) 

N  -  number  of  Items 

Simple  Averages 

The  first  error  measures  considered  were  simple  averages  of  traditional 


measures . 


1  N  1 

HAD  -  J  r  i-  I  J  E 

M  i-1  \  j »rt 


ij 


(Mean  Absolute  Demand) 


1  N  1 

USE  -  £  I  l 

“  i.i ’'i 


<Ei/ 


(Mean  Square  Error) 


1  N  1 
BIAS-i  I  i 

"  i-1  V 

(Average  Error) 


i  F 


'U 


P«rc ent  Error  Measures 


The  simple  averages  give  more  weight  to  items  with  higher  demand 
quantity.  Since  the  items  were  stratified  into  homogeneous  classes  it  was 
desirable  to  give  equal  weights  to  each  item  in  the  class,  hence  the  following 
percent  error  measures  were  considered. 


AVG  X  of  Forecast  -  — 


AVG  X  of  Actual 


AVG  X  of  Both 


1 

N 


1 

N 


;  * 
i-i  ^i 

;  * 
i-i 

i-1  ^i 


Relative  Error  Measures 


I 

^Fi 

1J 

c 

!eM| 

5tFi 

xij 

1  £  ! 

t 

^Fi 

1  (5 

2 

were 

quite  ' 

100 


x  100 


+  x  ) 
1J' 


x  100 


of  the  percent  error  measure*  did  not  reflect  the  steady  state  demand  of  the 
item  hence  the  following  relative  measure*  were  considered. 


RELATIVE  BIAS 


RELATIVE  MAD 


RELATIVE  MSE 


I  r  E 

N  i-1  ^i  lft  AVGi 


1  '  it 

i-1  ^ 


N 


'111 


AVG, 


i  J  i-  Z  Ibtl) 

s  i-i  ti  5 <».  *vci 
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6.  INVENTORY  MEASURES 


An  overforecast  error  In  predicting  demand  Impacts  the  inventory  control 
system  differently  than  does  an  underforecast.  Overforecasts  result  in  carrying 
too  much  stock  and  increases  the  possibility  of  being  stuck  with  obsolete 
items ,  whereas  underforecasts  increase  the  possibility  of  not  satisfying  a 
customer's  orders  and  in  the  case  of  the  Army  may  reduce  the  readiness  of  a 
weapon  system.  Since  there  is  not  a  natural  tradeoff  between  these  two  types 
of  errors,  the  following  separate  measures  were  developed. 


Notation 


,  th 


For  the  given  i  demand  series  and  its  corresponding  ^r^^ 

requisition  series  (the  number  of  requisitions  at  quarter  j) 


i«  Yd0';.!  *i.»j 


"i.fj 


be  the  demands  over  i  periods  from  time  t 

and  D  (O  .  the  forecast 
1  ,  c 

ba  the  number  of  requisitions  over  i  periods 


from  time  t 


EL.  U)  -  <D  (t)  -  D  ..(»)>  -  z  (x 

l.t  l,t  j.! 

errors  ovsr  lead  time  1 
UPj  ■  unit  pries  of  the  I1*1  item 


xi,t+j)  be  the 


Uverforecast  Measure 


Let  0F(1)  -  Z  nu. 


EL  (8) 

_ tqJ. 


.0 


Yi<8) 


UP, 


-  z  max  ^  EL  ,  (8),o\-  UP 


estimate  the  cost  of  the  extra  stock  purchased  for  periods  of  eight 
quarters  for  the  itlT  item. 
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Then 


OF  = 


N 

l  OF(i) 

i»l _ _ 

N 

Z  UP  Z  D  (8) 
i=l  j<F1  -1 


x  100 


is  an  estimate  of  the  percent  of  the  total  dollar  demand  npenf-  »xr^n  stock.* 
The  base  period  of  8  is  used  to  indicate  the  long  term  effect  of  an 
over forecast . 


Underforecast  Measure 


Similarily  let 

UF(i)  =  Z  max 
jtFj 


EL11 (PUT) 
D±j (PLT) 


R^CPLT) 


be  an  estimate  of  the  number  of  requisitions  not  satisfied  in  the  procurement 
lead  time,  PLT,  for  the  ith  item,  i.e.  if  demand  is  underforecasted,  say  by  20", 
then  it  is  implied  that  20%  of  the  requisitions  will  not  be  satisfied.  The 
reason  requisitions  are  used  instead  of  demand  quantity  is  because  of  DoD  policy 
( 1'oDI  4140.39]  . 


Then 


N 

Z  UF(i) 

UF  =  - -  x  100 

N 

I  (R  (PLT)) 

i-1  j€F±  3 


is  an  estimate  of  the  percent  of  the  total  requisition  not  satisfied  over  all 
the  items. 

The  procurement  lead  time  of  the  item  is  the  quickest  time  stock  could 
be  replenished  after  a  new  order  is  placed.  Hence,  in  an  underforecast  situation, 
the  reorder  point  will  probably  be  crossed  within  a  PLT,  and  a  new  forecast 


This  is  extra  stock  for  the  forecast  period  and  the  cost  estimate  associated 
with  it  does  not  consider  its  utility  as  safety  stock  for  future  under forecasts . 
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7.  SIMULATION  ANALYSIS 

The  third  way  to  evaluate  the  forecast  algorithms  is  to  observe  their 
performance  in  the  Army's  wholesale  management  system.  This  is  done  by  using 
the  DoDI  4140.39  Simulator  which  was  designed  to  compare  the  performance  of 
alternative  policies  within  this  management  system.  (Details  of  this  simulation 
may  be  found  in  [l],  [2],  and  14].)  The  basic  operational  sequence  of  the  simu¬ 
lator  can  best  be  described  from  the  following  excerpt  from  [1], 

The  demand  history  for  a  given  item  is  fed  both  to  the  forecast  algorithm 
process,  which  computes  the  demand  forecast  for  each  period;  and  to  the  asset 
monitor,  which  reduces  on-hand  stock  -  thus  possibly  triggering  an  order  for 
replenishment  -  and  computes  the  final  performance  estimates.  The  demand  fore¬ 
casts  are  fed  to  the  inventory  model,  which  computes  the  reorder  point  needed 
for  asset  monitoring  and  the  order  quantity  needed  when  orders  are  placed. 

The  final  performance  estimates  are  averaged  over  the  simulated  time  for  each 
item,  and  then  aggregated  across  all  items.  This  is  done  separately  for  each 
forecasting  algorithm. 

The  method  used  to  compare  policies  is  to  plot  a  cost-performance  curve 
for  each  rule  being  considered.  These  curves  are  traced  through  several  "X" 
points  where  the  lambda  (X)  value  reflects  the  cost  of  a  backorder.  (The 
lambda  value  is  set  prior  to  a  simulation  run) .  An  increase  in  X  results  in 
an  increase  in  inventory  cost  and  an  improvement  in  performance,  hence  X  can 
be  considered  as  a  control  knob  in  determining  various  points  on  the  curve. 

A  typical  graph  comparing  two  policies  Is  shown  below: 


(Note:  The  "Inventory  Cost"  as  shown  on  the  graph  includes  the  cost  to  order 

and  the  cost  to  hold  averaged  over  all  the  items  and  the  number  of  simulated 
years.  The  performance  measure,  "Average  Days  Wait,"  is  a  weighted  average 
of  the  average  wait  per  requisition  per  item.) 
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8.  COMPARISON  OF  ERROR  MEASURES 

It  was  hoped  that  a  simple  statistical  error  measure  could  be  found  that 
ranked  the  forecast  algorithms  according  to  their  performance  in  an  inventory 
management  system.  By  so  doing,  both  theoretical  and  empirical  guidance  would 
be  given  in  the  development  of  optimal  forecasts.  Currently  such  measures  as 
mean  squared  error  and  mean  absolute  deviation  are  minimized  but  these  criteria 
do  not  necessarily  guarantee  optimal  performance  for  inventory  management  as 
shown  in  this  section. 

The  following  is  a  list  of  observations  made  from  a  sequence  of  experiments 
comparing  error  measures  while  using  the  forecast  algorithms  introduced  in 
Section  4.  Tables  found  in  the  Appendix  are  referenced  after  each  observation. 

Initial  Observations: 

a.  The  simple  averages  MAD,  MSE,  and  BIAS  gave  more  weight  to  higher 
demand  items.  This  was  confirmed  by  reviewing  individual  item  contribution  to 
the  aggregation.  Refer  to  Appendix  Al. 

b.  The  average  percent  error  for  both  forecast  and  actual  yielded 
inconsistencies  between  their  1  quarter  and  4  quarter  measures  and  also  between 
themselves.  This  was  felt  to  be  due  to  the  variability  of  their  denominators. 

Refer  to  Appendix  A2. 

c.  Inconsistency  still  plagued  the  rankings  of  the  relative  error 
measures.  Refer  to  Appendix  A3. 

d.  Simulation  results  did  not  correspond  with  statistical  results 

where  a  consensus  was  used  to  rank  the  statistical  measures.  Refer  to  Appendix  A4 . 

Procedural  Changes 

Based  on  the  problems  between  the  statistical  measures  plus  the  desire 
to  predict  performance  In  an  inventory  setting,  the  following  procedural  changes 
were  made  to  the  method  of  computing  the  statistical  error  measures. 

a.  Forecast  only  after  a  non-zero  demand;  this  is  the  only  time  a 
reorder  point  may  be  triggered  and  where  the  forecast  is  actually  used 
(alternative  would  be  periodic  review  which  wasn't  considered). 

b.  Use  only  the  item's  PLT  as  a  forecast  horizon;  again  this  more 
clearly  corresponds  to  the  inventory  model. 

c.  Use  a  constant,  8  quarter  PLT,  for  the  basis  of  the  overforecast 
error  measure. 
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Additional  Observations : 


Applying  the  above  process  changes  to  both  the  statistical  and  inventory 
error  measures  and  to  the  nine  stratification  classes  (Section  3)  and  comparing 
these  results  with  the  simulator  yielded  the  tables  in  Appendix  A5.  (Note  the 
large  positive  bias  in  many  cases.)  The  following  inferences  were  drawn: 

a.  Limiting  the  algorithms  to  those  methods  not  exhibiting  a  large 
positive  bias,  several  comments  can  be  made. 

(1)  The  simulator  favored  those  methods  with  small  positive 
bias.  Additional  experiments  confirmed  this.  (Forecasts  were  increased  on 
methods  that  were  unbiased  and  the  simulated  performance  improved;  refer 

to  Appendix  A6) . 

(2)  The  inventory  measure  for  underforecast  compared  well  with 
the  simulator  but  this  measure  can  be  made  to  converge  to  zero  by  making  over¬ 
forecasts;  hence  the  need  to  limit  the  positive  bias. 

b.  Consistency  improved  between  the  ranks  of  the  various  statistical 
measures,  particularly  REL  MSE  and  REL  MAD. 

Simulation  Experimentation: 

Experimentation  with  the  simulator  revealed  the  following: 

* 

a.  Removing  the  phased  deliveries  and  the  R  and  Q  constraints  eliminated 
the  simulator's  preference  for  positive  bias  forecasts.  (Refer  to  A7) 

b.  The  simulator  performance  was  sensitive  to  individual  items  with  high 
demand  whereas  the  error  measures  were  designed  to  give  equal  weight  to  each 
item.  For  an  example  refer  to  Appendix  A8. 

Final  Adjustments: 

In  a  final  effort  to  correlate  the  statistical  measures  with  the  simulator 
the  following  adjustments  were  made: 

a.  Screen  from  the  simulation  results  those  items  that  carried  extreme 
high  cost  or  poor  performance  of  specific  algorithms.  (These  items  would  be 
analyzed  separately) . 


In  actual  practice  the  R  and  Q  decision  parameters  are  constrained  to  make  them 
less  sensitive  to  unreasonable  fluctuations  of  the  data.  Phased  deliveries  are 
used  to  spread  delivery  over  several  months  after  the  PLT.  Removing  both  of 
these  conditions  made  the  simulator  more  consistent  to  the  model  used  to  optimize 
the  parameters. 
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b.  For  use  with  the  simulator  and  error  measures,  use  a  simple  8 
quarter  average  for  the  forecast  If  the  Item  had  been  inactive  for  year  prior 
to  the  demand  triggering  the  forecast;  this  would  handle  the  migration  of  an 
item  from  an  active  strat  class  to  an  inactive  one  without  unduly  penalizing 
the  algorithm  which  would  normally  work  well  in  an  active  class  and  does  poorly 
in  the  less  active  class. 

Final  Observations: 

The  results  of  these  changes  for  the  five  active  stratification  classes 
are  in  Appendix  A9.  Looking  at  the  rank  statistics  the  following  inferences 
can  be  made: 

a.  For  the  most  part,  the  ranks  between  the  1  quarter  and  PLT  fore¬ 
cast  horizons  agreed. 

b.  The  ranks  between  the  three  statistical  measures  (MAD,  REL  MAD, 

REL  MSE)  were  more  consistent  than  before. 

c.  There  were  still  some  inconsistencies  between  the  statistical 
measures  ana  the  simulator. 


9.  SURROGATE  SIMULATOR 

Since  the  underforecast  measure  compared  well  with  the  simulator,  when 
the  overforecasts  were  constrained,  the  following  method  was  used  to  measure 
the  combined  effects  of  both  over  and  underforecasts. 

The  underforecast  measure  is  used  as  a  performance  indicator.  It  is  an 
estimate  of  the  percent  of  the  total  number  of  requisitions  not  satisfied  while 
using  a  specific  forecast  method. 

Similarly  the  overforecast  measure  is  used  as  a  cost  indicator.  This  is 
an  estimate  of  the  percent  of  the  total  dollar  demand  spent  on  extra  stock 
not  used  to  satisfy  demands. 

By  adjusting  the  forecast  (adding  or  subtracting  a  percent  of  the  forecast) 
to  represent  the  impact  of  adding  safety  stock  (negative  or  positive),  various 
performance  levels  are  attained.  With  each  level  of  performance  a  corresponding 
cost  is  computed  yielding  a  cost  vs  performance  curve  for  each  forecast  method. 


Curves  closest  to  the  X-Y  axis  represent  better  performance  or  lower  cost. 

To  test  this  concept,  the  three  strat  classes  ST9,  ST8,  and  ST7  were  used. 
Analysis  of  the  simulation  results  revealed  that  of  the  five  forecast  algorithms 
tested  only  MOVD  performed  differently  from  the  others.  This  occurred  in 
classes  ST9  and  ST8  where  MOVD  performed  worse  than  the  others. 

Using  the  surrogate  measure  with  the  three  strat  classes,  again  MOVD 
appeared  to  be  the  worse  performing  algorithm  in  ST9  and  ST7  strat  classes. 

Refer  to  Appendix  A10  for  results.  To  date  this  surrogate  simulation  method 
has  had  the  most  consistent  performance  when  compared  with  the  simulator. 
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10.  INACTIVE  ITEMS 


A  major  problem  in  statistically  comparing  different  forecast  algorithms 
using  series  with  many  zero  observations  is  that  traditional  error  measures  give 
substantial  weight  to  deviations  from  these  zero  modal  values.  Consequently, 
these  measures  favor  algorithms  yielding  routinely  small  forecasts  and,  in 
fact,  will  probably  most  favor  the  degenerate  algorithm  which  forecasts 
zero  all  the  time. 

Appendix  A3  of  [3]  describes  results  from  an  IRO  study  that  supports 
the  above  remarks.  In  particular,  it  demonstrates  that  MAD  did  favor  the 
degenerate  forecasts  of  zero.  Additional  work  done  on  various  other  studies 
at  IRO  also  support  these  findings.  Appendix  All  of  this  report  gives  additional 
credence  to  this  claim  using  samples  of  low  demand  items  from  the  subject  data 
base. 

There  are  also  problems  with  using  the  simulator  for  inactive  items. 

The  simulator  doesn't  adequately  duplicate  the  Army's  policy  of  managing  low 
demand  items;  details  of  which  will  not  be  discussed  here.  A  problem  which 
is  more  cumbersome  to  resolve  deals  with  the  way  the  simulator  distributes 
the  quarterly  data  into  32  periods  per  year.  This  "data  enrichment"  procedure 
was  not  designed  to  handle  sparse  demands  and  severely  distorts  the  actual 
series.  [9] 

The  only  resolution  to  this  problem  to  date  has  been  to  use  the  statistical 
method  of  Section  9.  A  further  study  by  IRO  will  deal  with  the  management  of 
low  demand  items  and  at  that  time  a  simulator  specifically  designed  for  these 
items  will  be  developed. 
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11.  CONCLUSION 


By  showing  empirical  anomalies  between  the  various  error  measures 
this  paper  has  brought  to  light  a  question  which  is  often  overlooked  when 
addressing  a  forecast  problem,  i.e.,  what  is  being  optimized?  There  is  much 
in  the  literature  on  minimizing  the  one  period  ahead  mean  squared  error,  but 
little  has  been  said  about  errors  over  a  long  term  horizon  and  their  impact 
on  the  working  environment . 

The  measure  given  in  Section  9  is  an  alternative  to  the  classical  error 
measures  which  relates  errors  to  performance  in  an  inventory  management 
system.  This  method  is  simpler  to  program  and  less  expensive  to  run  then  a 
detailed  simulation  and  its  results  have  compared  well  with  the  IRO  inventory 
simulator. 
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APPENDIX  A1 


INDIVIDUAL  ITEM  MEASURES 


DATA:  HDVDYN  y  N  - 
FORECAST  METHOD:  TRIG 
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APPENDIX  A3 


INCONSISTENCY  WITH  REL  ERROR 


DATE:  HDVDYN  ?  ~  ^ 

FORECAST  METHOD:  Seven  Different 

COMMENTS:  Variability  between  ranks  of  Relative  Error  Measures. 
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APPENDIX  AS 


STATISTICAL  AND  INVENTORY  ERROR  MEASURES  USING  PROCESS 
CHANCES  TO  BETTER  EMULATE  INVENTORY  MANAGEMENT 

(Actual  Values  and  Comparitive  Ranks  Along  With  Simulation  Ranks) 


DATA:  STRAT  CLASSES  ST4,  ST5,  ST6,  ST7,  ST8 

FORECAST  METHODS:  TRIG,  MEDIAN,  1794,  KALMAN 
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APPENDIX  A6 


EFFECTS  OF  ADDING  BIAS  TO  FORECAST  METHODS  USING  SIMULATOR 
WITH  PHASED  DELIVERIES  AND  R-Q  CONSTRAINTS 


DATA:  STRAT  CLASSES  ST4  AND  ST8 

FORECAST  METHOD:  1794  ADJUSTED  FOR  BIAS 
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APPENDIX  A7 


EFFECTS  OF  ADDING  BIAS  TO  FORECAST  METHOD  USING  SIMULATOR 
WITHOUT  BOTH  PHASED  DELIVERIES  AND  R-Q  CONSTRAINTS 


DATA:  STRAT  CLASS  ST4  AND  ST8 

FORECAST  METHOD:  1794  ADJUSTED  FOR  BIAS 
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APPENDIX  A8 


EXHIBIT  OF  SIMULATOR  SENSITIVITY  TO  A  SINGLE  ITEM 


DATA:  STRAT  CLASS  ST8 

FORECAST  METHOD:  KALMAN  FILTER  WITH  AND  WITHOUT  ITEM  it 6 
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APPENDIX  A9 


FINAL  COMPARISON  OF  STATISTICAL  ERROR  MEASURES  AND  SIMULATION 

RESULTS 

STRAT  CLASSES:  ST4,  ST5,  ST7 ,  ST8,  ST9 

ERROR  MEASURES:  ERROR  (BIAS),  REL  ERROR,  MAD,  REL  MAD,  REL  MSE,  SIMULATION 

RANKINGS 
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APPENDIX  AIO 


GRAPHS  COMPARING  GOST  VS  PERFORMANCE  RELATIONSHIPS  USING 
STATISTICAL  (SURROGATE  SIMULATOR)  AND  SIMULATED 

MEASURES 

STRAT  CLASSES:  ST9 ,  ST8 ,  ST7 
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STATISTICAL  ERROR  MEASURES  USING  A  LOW  DEMAND  DATA  BASE 


42 


r 


i 


! 

i 

REFERENCES 

1.  Cohen,  Martin,  "Alpha  4140.39  Simulator,"  AMC  Inventory  Research  Office, 

May  1973,  AD762348. 

2.  Cohen,  Martin,  "Demand  Forecasting  With  Program  Factors,"  AM C  Inventory  ! 

Research  Office,  September  1975,  ADA017858. 

3.  Ootwals,  E.  P.  &  A.  Hutchison,  "Use  of  End  Item  Age  in  Demand  Forecasting," 

PRC  Inventory  Research  Office,  December  1976,  ADA037585.  > 

I 

4.  Gotwals,  E.  P.  and  D.  Orr,  "Integrated  Forecasting  Techniques  for  Secondary 
Item  Classes  -  Part  1,  Active  Items,"  US  Army  Inventory  Research  Office, 

September  1980,  ADA097024. 

5.  Cotwals,  E.  P.,  "Integrated  Forecasting  Techniques  fur  Secondary  Item 
Classes  -  Part  II,  Inactive  Items,"  US  Army  Inventory  Research  Office, 

September  1980,  ADA097025. 

6.  Meng,  Daniel,  "Application  of  Dynamic  Mean  to  Demand  Forecast,"  System 
Analysis  Division,  Tank  Automotive  Materiel  Readiness  Command,  September 
1977,  Report  No.  SA-77-11. 

7.  Orr,  Donald,  "Demand  Forecasts  Using  Process  Models  and  Item  Class  Parameters: 
Application  of  Ancillary  Variables,"  DRC  Inventory  Research  Office,  April 
1976,  ADA026081. 

8.  Orr,  Donald,  "Kalman  and  Moving  Average  Filters  for  Forecasting:  Systematics 
of  Demand  Processes  and  Extension,"  DRC  Inventory  Research  Office,  October 
1976,  ADA032476. 

9.  Orr,  Donald  and  Alan  Kaplai^  "Data  Enrichment  for  Simulation,"  Simulation. 

June  1976,  177-183. 


44 


■  ■  V 


* 


NO 


DISTRIBUTION 


COPIES 

1  Deputy  Under  Sec’y  of  the  Army,  ATTN:  Office  of  Op  Reach 

Headquarters,  US  Army  Materiel  Development  &  Readiness  Command 
1  DRCPS-P ,  ATTN:  Ms.  Lamb 

1  DRCMM-R 

1  DRCMM-S 

1  DRCMM-RS 

1  DRCMM-M 

1  DRCMM-E 

1  DRCRE 

1  Dep  Chf  of  Staff  for  Logistics,  ATTN:  DALO-SMS,  Pentagon, 

Wash.,  DC  20310 

1  Commandant,  US  Army  Logistics  Mgt  Center,  Ft.  Lee,  VA  23801 

1  Office,  Asst  Sec'y  of  Defense,  ATTN:  MRA&L-SR,  Pentagon, 

Wash.,  DC  20310 

Defense  Logistics  Studies  Info  Exchange,  DRXMC-D 
Defense  Technical  Information  Center,  Cameron  Station,  Alexandria, 

VA  22314 

Commander,  USA  Armament  Materiel  Readiness  Cmd,  Rock  Island,  IL  61299 

1 _  ATTN:  DRSAR-MM 

1 _  ATTN:  DRSAR-SA 

Commander,  USA  Communications-Electronics  Cmd,  Ft.  Monmouth,  NJ  07703 

1 _  ATTN:  DRSEL-MM 

1 _  ATTN:  DRSEL-PL-SA 

Commander,  USA  Missile  Command,  Redstone  Arsenal,  AL  33898 

1 _  ATTN:  DRSMI-S 

1 _  ATTN:  DRSMI-D 

Commander,  USA  Troop  Support  &  Aviation  Materiel  Readiness  Command, 

St.  Louis,  MO  63120 

1 _  ATTN:  DRSTS-SPF 

_1 _  ATTN:  DRSTS-SPS 

1 _  ATTN:  DRSTS-BA 

Commander,  US  Army  Tank-Automotive  Command,  Warren,  MI  48090 

1 _  ATTN:  DRSTA-F 

1 _  ATTN:  DRSTA-S 

_1 _  Commander,  US  Army  Armament  Research  &  Development  Cmd,  ATTN:  DRDAR-SL , 

Dover,  NJ  07801 

1 _  Commander,  US  Army  Aviation  Research  &  Development  Cmd,  4300  Goodfellow 

Blvd ,  St.  Louis,  M0  63120 

1 _  Commander,  US  Army  Aviation  Research  &  Development  Cmd,  ATTN:  DRDAV-BD 

(Marl  Glenn),  4300  Goodfellow  Blvd,  St.  Louis,  MO  63120 

1 _  Commander,  US  Army  Electronics  Research  &  Development  Command, 

ATTN:  DRDEL-ST-SA,  Rm  2D315,  Bldg  2700,  Ft.  Monmouth,  NJ  07703 

1 _  Commander,  US  Army  Mobility  Equipment  Research  &  Development  Cmd, 

ATTN:  DRDME-0,  Ft.  Belvoir,  VA  22060 

1 _  Commander,  US  Army  Natick  Research  &  Development  Command, 

ATTN:  DRXNM-0,  Natick,  MA  01760 

1 Commander,  US  Army  Logistics  Center,  Ft.  Lee,  VA  23801 

1 Commander,  US  Army  Logistics  Evaluation  Agency,  New  Cumberland 

Army  Depot,  New  Cumberland,  PA  17070 

45 


COPIES 


1 

1 


1 


1 


1 


1 


1 


I 

1 

1 


1 


1 


1 


I 

1 


1 


1 

1 


I 

1 


1 


1 

1 

1 


1 


I 

1 


I 


1 


1 

1 

1 


Commander,  US  Army  Depot  Systems  Command,  Chambersburg,  PA  17201 
Commander,  US  Air  Force  Logistics  Cmd,  WPAFB,  ATTN:  AFLC/XRS, 

Dayton,  Ohio  45433 

US  Navy  Fleet  Materiel  Support  Office,  Nival  Support  Depot, 

Mechanicsburg,  PA  17055 

Mr.  James  Prichard,  Navy  SEA  Systems  Cmd,  ATTN:  PMS  3061,  Dept 
of  US  Navy,  Wash.,  DC  20362 

George  Washington  University,  Inst  of  Management  Science  &  Engr., 

707  22nd  St.,  N.W.,  Wash.,  DC  20006 
Naval  Postgraduate  School,  ATTN:  Dept  of  Opns  Anal,  Monterey, 

CA  93940 

Air  Force  Institute  of  Technology,  ATTN:  SLGQ,  Head  Quantitative 
Studies  Dept.,  Dayton,  OH  43433 
US  Army  Military  Academy,  West  Point,  NY  10996 

Librarian,  Logistics  Mgt  Inst.,  4701  Sangamore  Rd. ,  Wash. , DC  20016 
University  of  Florida,  ATTN:  Dept  of  Industrial  Systems  Engr., 

Gainesville,  FL  32601 

RAND  Corp.,  ATTN:  S.  M.  Drezner,  1700  Main  St.,  Santa  Monica, 

CA  90406 

US  Army  Materiel  Systems  Analysis  Activity,  ATTN:  DRXSY-CL, 

Aberdeen  Proving  Ground,  MD  21005 
Commander,  US  Army  Logistics  Center,  ATTN:  Concepts  &  Doctrine 
Directorate,  Ft.  Lee,  VA  23801 
ALOG  Magazine,  ATTN:  Tom  Johnson,  USALMC,  Ft.  Lee,  VA  23801 
Commander,  USDRC  Automated  Logistics  Mgt  Systems  Activity, 

P.O.  Box  1578,  St.  Louis,  MO  63188 
Director,  DARCOM  Logistics  Systems  Support  Agency,  Letterkenny 
Army  Depot,  Chambersburg,  PA  17201 
Commander,  Materiel  Readiness  Support  Activity,  Lexington,  KY  40507 
Director,  Army  Management  Engineering  Training  Agency,  Rock  Island 
Arsenal,  Rock  Island,  IL  61299 

Defense  Logistics  Agcy,  ATTN:  DLA-LO ,  Cameron  Sta,  Alexandria,  VA  22314 
Dep  Chf  of  Staff  (I&L),  HQ  USMC-LMP-2,  ATTN:  LTCJ  Sonneborn,  Jr., 

Wash.,  DC  20380 

Commander,  US  Army  Depot  Systems  Command,  Letterkenny  Army  Depot, 

ATTN:  DRSDS-LL,  Chambersburg,  PA  17201 
Logistics  Control  Activity,  Presidio  of  San  Francisco,  CA  94120 
HQ,  Dept  of  the  Army,  (DASG-HCL-P) ,  Wash.,  DC  20314 
Operations  Research  Center,  3115  Etcheverry  Hall,  University 
of  California,  Berkeley,  CA  94720 
Dr.  Jack  Muckstadt,  Dept  of  Industrial  Engineering  &  Operations 
Research,  Upson  Hall,  Cornell  University,  Ithaca,  NY  14890 
Dept  of  Industrial  Engineering,  University  of  Michigan,  Ann  Arbor,  MI  48104 
Scientific  Advisor,  ATCL-SCA,  Army  Logistics  Center,  Ft.  Lee,  VA 
23801 

Prof  Robert  M.  Stark,  Dept  of  Stat  4  Computer  Sciences, 

University  of  Delaware,  Newark,  DE  19711 
Prof  E.  Gerald  Hurst,  Jr.,  Dept  of  Decision  Science,  The  Wharton 
School,  University  of  Penna.,  Phila.,  PA  19104 
Logistics  Studies  Office,  DRXMC-LSO,  ALMC,  Ft.  Lee,  VA  23801 
Procurement  Research  Office,  DRXMC-PRO,  ALMC,  Ft.  Lee,  VA  23801 
Dept  of  Industrial  Engr.  &  Engr.  Management,  Stanford  University, 

Stanford,  CA  94305 


46 


Commander,  US  Army  Communications  Command,  ATTN:  Dr.  Forry, 

CC-LOG-LEO,  Ft.  Huachuca,  AZ  85613 
Prof  Harvey  M.  Wagner,  Dean,  School  of  Business  Adm,  University 
of  North  Carolina,  Chapel  Hill,  NC  27514 
DARCOM  Intern  Training  Center,  Red  River  Army  Depot,  Texarkana, 

Tx  75501 

Prof  Leroy  B.  Schwarz,  Dept  of  Management,  Purdue  University, 

Krannert  Bldg,  West  Lafayette,  Indiana  47907 
US  Army  Training  &  Doctrine  Command,  Ft.  Monroe,  VA  23651 
US  General  Accounting  Office,  ATTN:  Mr.  J.  Morris,  Rm  5840, 

441  G.  St.,  N.W.,  Wash.,  DC  20548 
Operations  &  Inventory  Analysis  Office,  NAVSUP  (Code  04A)  Dept 
of  Navy,  Wash.,  DC  20376 

US  Army  Research  Office,  ATTN:  Robert  Launer,  Math.  Div. , 

P.0.  Box  12211,  Research  Triangle  Park,  NC  27709 
Prof  William  P.  Pierskalla,  3641  Locust  Walk  CE,  Philadelphia, 

PA  19104 

US  Army  Materiel  Systems  Analysis  Activity,  ATTN:  DRXSY-MP, 

Aberdeen  Proving  Ground,  MD  21005 

Air  Force  Logistics  Management  Center,  Gunter  Air  Force  Station,  AL  36144 
ATTN:  AFLMC/LGY 
ATTN:  AFLMC/XRP,  Bldg.  205 

Engineer  Studies  Center,  6500  Brooks  Lane,  Wash.,  DC  20315 
US  Army  Materiel  Systems  Analysis  Activity,  ATTN:  Mr.  Herbert  Cohen, 
DRXSY-MP,  Aberdeen  Proving  Ground,  MD  21105 
Dr.  Carl  Weisman,  C.A.C.I.,  1815  North  Fort  Myer  Drive,  Arlington, VA  22209 
Commander,  US  Army  Missile  Cmd,  ATTN:  Ray  Dotson,  DRSMI-DS,  Redstone 
Arsenal,  AL  35898 

Prof  Peter  Kubat ,  Graduate  School  of  Mgmt,  University  of  Rochester, 
Rochester,  NY  14627 

Prof  Barney  Bisslnger,  The  Pennsylvania  State  University, 

Middletown,  PA  17057 

Prof  Gary  D,  Scudder,  Mgmt  Sciences  Dept,  College  of  Bus  Adm,  University 
of  Minnesota,  271  19th  Ave.,  South,  Minneapolis,  MN  55455 


47 


